Manually reviewing certificates in VMware Endpoint Certificate Store for vSphere 6.x and 7.x

Products

• VMware vCenter Server

Issue/Introduction

This article provides information on how to manually reviewing the Certificate Authority (CA) signed SSL certificates in a vSphere 6 or 7 environment. In vSphere 6 and 7, certificates generated by the VMware Certificate Authority (VMCA) can be monitored through the vSphere Web Client. For more information, see the View vCenter Certificates with the vSphere Web Client section in the Platform Services Controller Administration Guide.

Note: You will need to manage your own certificate validity if you are using your own Private Key Infrastructure (PKI) in your environment.

This article uses the vecs-cli command to list certificates stored in the VMware Endpoint Certificate Store (VECS) as well as references the individual keystores used by vSphere. Before proceeding, familiarize yourself by reviewing the Where vSphere Uses Certificates and vecs-cli Command Reference section of the vSphere Authentication Guide.

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 6.7.x
VMware vCenter Server 6.5.x

Resolution

• Reviewing stored certificates for Windows
• Reviewing stored certificates for vCenter Server Appliance
• Exporting stored certificates for Windows
• Exporting stored certificates for vCenter Server Appliance

Reviewing stored certificates for Windows:

Note: These steps are written using the default installation path of C:\ drive on Windows. If you are using a non-default installation path, these steps need to be modified.

1. Open an elevated command prompt. If an elevated command prompt is not used, the entry list commands fail with Win Error: access is denied.
2. Run this command to list all of the Keystores stored within VECS. This command can be run from a vCenter Server or a Platform Services Controller.
   
   "C:\Program Files\VMware\vCenter Server\vmafdd\"vecs-cli store list
   
   This will output one of these lists depending on what node this command is performed on.

   For External Platform Services Controller

   MACHINE_SSL_CERT
   TRUSTED_ROOTS
   TRUSTED_ROOT_CRLS
   machine

   For vCenter Server with external Platform Services Controller or embedded Platform Services Controller

   MACHINE_SSL_CERT
   TRUSTED_ROOTS
   TRUSTED_ROOT_CRLS
   machine
   vpxd
   vpxd-extension
   vsphere-webclient
   SMS
    
3. To review the individual Keystores, use these examples to list the certificate stored:
   
   "C:\Program Files\VMware\vCenter Server\vmafdd\"vecs-cli entry list --store TRUSTED_ROOTS --text | more
   "C:\Program Files\VMware\vCenter Server\vmafdd\"vecs-cli entry list --store MACHINE_SSL_CERT --text | more
   "C:\Program Files\VMware\vCenter Server\vmafdd\"vecs-cli entry list --store machine --text | more
   "C:\Program Files\VMware\vCenter Server\vmafdd\"vecs-cli entry list --store vpxd --text | more
   "C:\Program Files\VMware\vCenter Server\vmafdd\"vecs-cli entry list --store vpxd-extension --text | more
   "C:\Program Files\VMware\vCenter Server\vmafdd\"vecs-cli entry list --store vsphere-webclient --text | more
   "C:\Program Files\VMware\vCenter Server\vmafdd\"vecs-cli entry list --store SMS --text | more
   "C:\Program Files\VMware\vCenter Server\vmafdd\"vecs-cli entry list --store TRUSTED_ROOT_CRLS --text | more
   
   Note: Press q to leave the session.
    
4. Review the certificates displayed.
   
   Notes:

• When reviewing the MACHINES_SSL_CERT or any of the Solution User stores, take note of the X509v3 extensions, particularly Key Usages, Validity, and Subject Alternate Name. For customers who upgraded to vSphere 6, the MACHINE_SSL_CERT will now be the certificate previously used for the vCenter Server.
• When reviewing the TRUSTED_ROOTS store, take note of the X509v3 extensions, particularly the Key Usage Certificate Sign and Validity. If the Certificate Sign Key Usage is missing, the VMCA is unable to sign and provision certificates thus causing installation and certificate regeneration failures
• You might see below Stores as well in some situations depending on vCenter build, use the vecs-cli commands mentioned above to list the certificates stored in these stores:
  • BACKUP_STORE
  • data-encipherment
  • KMS_ENCRYPTION

Reviewing stored certificates for vCenter Server Appliance:

1. Open an SSH session to the vCenter Server Appliance. Log in as root. Switch to using a BASH shell session by using this command:
   
   shell.set --enabled true
   
   shell
    
2. Run this command to list all of the Keystores stored within VECS. This command can be run from a vCenter Server or a Platform Services Controller.
   
   /usr/lib/vmware-vmafd/bin/vecs-cli store list
   
   This will output one of the following lists depending on what node this command is performed on.

   External Platform Services Controller

   MACHINE_SSL_CERT
   TRUSTED_ROOTS
   TRUSTED_ROOT_CRLS
   machine

   For vCenter Server Appliance with external Platform Services Controller or embedded Platform Services Controller
   
   MACHINE_SSL_CERT
   TRUSTED_ROOTS
   TRUSTED_ROOT_CRLS
   machine
   vpxd
   vpxd-extension
   vsphere-webclient
   SMS

3. In order to review the individual Keystores, use these examples to list the certificate stored:
   
   /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | less
   /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text | less
   /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store machine --text | less
   /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd --text | less
   /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd-extension --text | less
   /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vsphere-webclient --text | less
   /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store SMS --text | less
   /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOT_CRLS --text | less
   
   Note: Press q to leave the session.
    
4. Review the certificates displayed.
   
   Notes:

• When reviewing the MACHINES_SSL_CERT or any of the Solution User stores, take note of the X509v3 extensions, particularly Key Usages, Validity, and Subject Alternate Name For customers who upgraded to vSphere 6, the MACHINE_SSL_CERT will now be the certificate previously used for the vCenter Server.
• When reviewing the TRUSTED_ROOTS store, take note of the X509v3 extensions, particularly the Key Usage Certificate Sign and Validity. If the Certificate Sign Key Usage is missing, the VMCA is unable to sign and provision certificates thus causing installation and certificate regeneration failures.
• You might see below Stores as well in some situations depending on vCenter build, use the vecs-cli commands mentioned above to list the certificates stored in these stores:
  • BACKUP_STORE
  • data-encipherment
  • KMS_ENCRYPTION

Exporting stored certificates for Windows:

Note: These steps are written using the default installation path of C:\ drive on Windows. If you are using a non-default installation path, these steps need to be modified.

1. Open an elevated command prompt. If an elevated command prompt is not used, the entry list commands fail with Win Error: access is denied.
2. Run these commands to list all of the Keystores stored within VECS. These commands can be run from a vCenter Server or a Platform Services Controller. Create directory "C:\Certificates" before proceeding with below steps. 
   
   "%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store <stored name> --alias <alias name> --output c:\certificates\<certificate usage name>.crt
   "%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getkey --store <stored name> --alias <stored name> --output c:\certificates\<certificate usage name>.key
   
   Users have the option to output one of the following store's pair depending on what node this command is performed on.

   For External Platform Services Controller

   MACHINE_SSL_CERT
   TRUSTED_ROOTS
   TRUSTED_ROOT_CRLS
   machine

   For vCenter Server with external Platform Services Controller or embedded Platform Services Controller
   
   MACHINE_SSL_CERT
   TRUSTED_ROOTS
   TRUSTED_ROOT_CRLS
   machine
   vpxd
   vpxd-extension
   vsphere-webclient
   SMS

3. To review the individual Keystores, use these examples to list the certificate stored:
   
   "%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output c:\certificates\machine_ssl.crt
   "%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output c:\certificates\machine_ssl.key
   "%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output c:\certificates\vpxd-extension.crt
   "%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output c:\certificates\vpxd-extension.key
   "%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vpxd --alias vpxd --output c:\certificates\vpxd.crt
   "%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getkey --store vpxd --alias vpxd --output c:\certificates\vpxd.key
   "%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store machine --alias machine --output c:\certificates\machine.crt
   "%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getkey --store machine --alias machine --output c:\certificates\machine.key
   "%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output c:\certificates\vsphere-webclient.crt
   "%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getkey --store vsphere-webclient --alias vsphere-webclient --output c:\certificates\vsphere-webclient.key

Exporting stored certificates for vCenter Server Appliance:

1. Open an SSH session to the vCenter Server Appliance. Log in root. Switch to using a BASH shell session by using the command:
   
   shell.set --enabled true
   
   shell
    
2. Create the export location directory by running this command "mkdir /certificate".
3. Run these commands to export the Key and Certificate pairs stored within VECS one by one. These commands can be run from a vCenter Server or a Platform Services Controller.
   
   /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store <stored name> --alias <alias name> --output /certificate/<certificate usage name>.crt
   /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store <stored name> --alias <stored name> --output /certificate/<certificate usage name>.key
   
   Users have the option to output one of the following store's pair depending on what node this command is performed on.

   External Platform Services Controller

   MACHINE_SSL_CERT
   TRUSTED_ROOTS
   TRUSTED_ROOT_CRLS
   machine

   For vCenter Server Appliance with external Platform Services Controller or embedded Platform Services Controller

   MACHINE_SSL_CERT
   TRUSTED_ROOTS
   TRUSTED_ROOT_CRLS
   machine
   vpxd
   vpxd-extension
   vsphere-webclient
   SMS

4. Use these commands as examples for exporting the stored pairs.
   
   /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /certificate/Machine_SSL.crt
   /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /certificate/Machine_SSL.key
   /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt
   /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key
   /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd --alias vpxd --output /certificate/vpxd.crt
   /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd --alias vpxd --output /certificate/vpxd.key
   /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store machine --alias machine --output /certificate/machine.crt
   /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store machine --alias machine --output /certificate/machine.key
   /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output /certificate/vsphere-webclient.crt
   /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vsphere-webclient --alias vsphere-webclient --output /certificate/vsphere-webclient.key

Additional Information

• VMware Skyline Health Diagnostics for vSphere - FAQ (81931)
• "Error in creating a new entry for __MACHINE_CERT in VECS Store MACHINE_SSL_CERT", Certificate Replacement with Custom Certificate fails on 6.x/7.x (70777)
• "Operation failed with error ERROR_INVALID_PARAMETER (87)" & "Failed to get vecs users and permissions" while upgrading from vCSA 6.5U2d to 6.7U1 (59508)
• "Unable to enumerate and validate the root certificates from the TRUSTED_ROOTS VECS store" pre-upgrade check error while upgrading to VC 6.7 (70902)
• "VasaServiceException: org.apache.axis2.AxisFault: certificate has expired", SMS Certificate Expiry Alarm after upgrading vCenter Server from 5.x to 6.x (2120105)
• How to replace the vSphere 6.0 Solution User certs with VMCA issued certs (2112281)

For information on renewing VMware certificates, see:

• Replacing the vSphere 6.0 Machine SSL certificate with a VMware Certificate Authority issued certificate (2112279)